Inside Observability with .NET 10

Table of Contents

1. Overview
2. Observability Fundamentals
3. .NET 10 Runtime Observability Enhancements
4. Built-in Metrics and Telemetry
5. OpenTelemetry Integration
6. .NET Aspire Observability Stack
7. Monitoring and Diagnostic Tools
8. Performance Profiling and Analysis
9. Distributed Tracing
10. Best Practices and Implementation Patterns
11. Real-World Scenarios
12. Troubleshooting and Debugging

Overview

Observability in .NET 10 represents a fundamental shift toward comprehensive application monitoring, diagnostics, and performance analysis. The new runtime introduces native instrumentation capabilities, enhanced metrics collection, and seamless integration with modern observability platforms.

This document provides an in-depth exploration of observability features in .NET 10, covering both runtime-level enhancements and the powerful observability stack provided by .NET Aspire.

Observability Fundamentals

The Three Pillars of Observability

1. Metrics

Quantitative measurements that provide insights into system performance and behavior:

• Counter metrics: Request counts, error counts, operation counts
• Gauge metrics: Memory usage, CPU utilization, active connections
• Histogram metrics: Request duration, response size distribution
• Summary metrics: Percentile calculations, aggregated statistics

2. Logs

Structured events that capture detailed information about application execution:

• Structured logging: JSON-formatted logs with consistent schemas
• Contextual information: Request IDs, user IDs, operation context
• Log levels: Debug, Information, Warning, Error, Critical
• Correlation: Cross-service log correlation and tracing

3. Traces

Distributed execution paths that show request flow across services:

• Spans: Individual operations within a trace
• Trace context: Propagation across service boundaries
• Baggage: Key-value pairs carried across spans
• Sampling: Intelligent trace collection strategies

Modern Observability Requirements

Cloud-Native Applications

• Microservices architecture: Multiple services requiring coordinated monitoring
• Container orchestration: Kubernetes and container-specific metrics
• Auto-scaling: Dynamic resource allocation monitoring
• Service mesh: Network-level observability and security metrics

Performance Engineering

• Real-time monitoring: Live performance dashboards and alerting
• Capacity planning: Resource utilization trends and forecasting
• Bottleneck identification: Performance hotspot detection
• User experience: End-user performance monitoring

.NET 10 Runtime Observability Enhancements

Native Instrumentation

Built-in Semantic Conventions

.NET 10 introduces native OpenTelemetry semantic conventions without requiring additional packages:

// Automatic instrumentation for HTTP requests
public class ApiController : ControllerBase
{
    [HttpGet("/api/products")]
    public async Task<IActionResult> GetProducts()
    {
        // Automatically generates:
        // - http.method = "GET"
        // - http.url = "/api/products"
        // - http.status_code = 200
        // - http.response_time_ms = execution_duration
        
        return Ok(await productService.GetProductsAsync());
    }
}

Activity and Span Enhancement

Improved Activity API with richer context and better performance:

using System.Diagnostics;

public class OrderProcessingService
{
    private static readonly ActivitySource ActivitySource = 
        new("OrderProcessing", "1.0.0");
    
    public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        
        // Automatic context propagation
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);
        
        try
        {
            var order = await CreateOrderAsync(request);
            activity?.SetTag("order.id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);
            
            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
}

Enhanced Metrics Collection

Kestrel Memory Pool Metrics

Real-time memory tracking for web server optimization:

// Automatic metrics collection
public class MemoryPoolMetrics
{
    // Collected automatically by .NET 10 runtime
    public static readonly Counter<int> MemoryPoolAllocations = 
        Meter.CreateCounter<int>("kestrel.memory_pool.allocations");
    
    public static readonly Gauge<long> MemoryPoolUsage = 
        Meter.CreateGauge<long>("kestrel.memory_pool.bytes_used");
    
    public static readonly Histogram<double> MemoryPoolReleaseLatency = 
        Meter.CreateHistogram<double>("kestrel.memory_pool.release_latency_ms");
}

Authentication and Authorization Metrics

Security operation monitoring with detailed insights:

public class AuthMetrics
{
    // Login attempt tracking
    public static readonly Counter<int> LoginAttempts = 
        Meter.CreateCounter<int>("auth.login.attempts");
    
    // Authentication success/failure rates
    public static readonly Counter<int> AuthenticationResults = 
        Meter.CreateCounter<int>("auth.authentication.results");
    
    // Token validation performance
    public static readonly Histogram<double> TokenValidationDuration = 
        Meter.CreateHistogram<double>("auth.token.validation_duration_ms");
    
    // Passkey authentication metrics
    public static readonly Counter<int> PasskeyOperations = 
        Meter.CreateCounter<int>("auth.passkey.operations");
}

Blazor-Specific Observability

Circuit Monitoring

Real-time Blazor Server circuit tracking:

public class BlazorMetrics
{
    // Active circuit count
    public static readonly Gauge<int> ActiveCircuits = 
        Meter.CreateGauge<int>("blazor.circuits.active");
    
    // Circuit connection state
    public static readonly Counter<int> CircuitStateChanges = 
        Meter.CreateCounter<int>("blazor.circuits.state_changes");
    
    // Interactive rendering performance
    public static readonly Histogram<double> RenderingDuration = 
        Meter.CreateHistogram<double>("blazor.rendering.duration_ms");
    
    // SignalR connection metrics
    public static readonly Gauge<int> SignalRConnections = 
        Meter.CreateGauge<int>("blazor.signalr.connections");
}

WebAssembly Diagnostics

Browser-based performance profiling:

// Browser DevTools integration
public class BlazorWasmDiagnostics
{
    public static void EnableBrowserProfiling()
    {
        // Automatic CPU sampling
        DiagnosticSource.StartCpuSampling();
        
        // Memory allocation tracking
        GC.StartConcurrentGCTracking();
        
        // Performance counter collection
        PerformanceCounters.EnableCollection();
    }
    
    public static async Task<DiagnosticData> ExtractDiagnosticsAsync()
    {
        return new DiagnosticData
        {
            CpuProfile = await DiagnosticSource.GetCpuProfileAsync(),
            MemoryDump = await GC.GetMemoryDumpAsync(),
            PerformanceCounters = PerformanceCounters.GetSnapshot()
        };
    }
}

Built-in Metrics and Telemetry

HTTP Request Metrics

Automatic Collection

.NET 10 automatically collects comprehensive HTTP metrics:

// Automatically generated metrics for each HTTP request
public class HttpMetrics
{
    // Request count by method and status code
    // http.server.requests{method="GET", status_code="200"}
    
    // Request duration histogram
    // http.server.request.duration{method="GET", route="/api/products"}
    
    // Request body size
    // http.server.request.body.size{method="POST", route="/api/orders"}
    
    // Response body size
    // http.server.response.body.size{method="GET", route="/api/products"}
    
    // Active requests gauge
    // http.server.active_requests{method="GET"}
}

Custom HTTP Metrics

Extending built-in metrics with application-specific data:

public class CustomHttpMetrics
{
    private static readonly Counter<int> ApiCallsByClient = 
        Meter.CreateCounter<int>("api.calls.by_client");
    
    private static readonly Histogram<double> BusinessOperationDuration = 
        Meter.CreateHistogram<double>("business.operation.duration_ms");
    
    public static void RecordApiCall(string clientId, string operation, double duration)
    {
        ApiCallsByClient.Add(1, new KeyValuePair<string, object>("client_id", clientId));
        BusinessOperationDuration.Record(duration, 
            new KeyValuePair<string, object>("operation", operation));
    }
}

Database Operation Metrics

Entity Framework Core Integration

Automatic database performance monitoring:

public class DatabaseMetrics
{
    // Query execution time
    public static readonly Histogram<double> QueryDuration = 
        Meter.CreateHistogram<double>("ef.query.duration_ms");
    
    // Connection pool metrics
    public static readonly Gauge<int> ConnectionPoolSize = 
        Meter.CreateGauge<int>("ef.connection_pool.size");
    
    // Failed query attempts
    public static readonly Counter<int> QueryErrors = 
        Meter.CreateCounter<int>("ef.query.errors");
}

// Usage in DbContext
public class ApplicationDbContext : DbContext
{
    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        optionsBuilder.EnableDetailedErrors()
                     .EnableSensitiveDataLogging(isDevelopment)
                     .AddInterceptors(new MetricsInterceptor());
    }
}

Memory and GC Metrics

Garbage Collection Monitoring

Real-time memory management insights:

public class MemoryMetrics
{
    // GC collection count by generation
    public static readonly Counter<int> GCCollections = 
        Meter.CreateCounter<int>("gc.collections");
    
    // Memory allocated per generation
    public static readonly Gauge<long> GenerationSize = 
        Meter.CreateGauge<long>("gc.generation.size_bytes");
    
    // Time spent in GC
    public static readonly Counter<double> GCTime = 
        Meter.CreateCounter<double>("gc.time_ms");
    
    // Large object heap size
    public static readonly Gauge<long> LohSize = 
        Meter.CreateGauge<long>("gc.loh.size_bytes");
}

OpenTelemetry Integration

Native Integration Benefits

Zero-Configuration Setup

.NET 10 provides built-in OpenTelemetry integration without external packages:

public class Program
{
    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        
        // Native OpenTelemetry - no additional packages needed
        builder.Services.AddOpenTelemetry()
            .WithMetrics(metrics =>
            {
                metrics.AddAspNetCoreInstrumentation()
                       .AddHttpClientInstrumentation()
                       .AddEntityFrameworkCoreInstrumentation();
            })
            .WithTracing(tracing =>
            {
                tracing.AddAspNetCoreInstrumentation()
                       .AddHttpClientInstrumentation()
                       .AddEntityFrameworkCoreInstrumentation();
            });
        
        var app = builder.Build();
        app.Run();
    }
}

Semantic Conventions

Standardized telemetry attributes following OpenTelemetry specifications:

// Automatic semantic conventions for HTTP operations
public class SemanticConventions
{
    // HTTP attributes
    public const string HttpMethod = "http.method";
    public const string HttpStatusCode = "http.status_code";
    public const string HttpUrl = "http.url";
    public const string HttpUserAgent = "http.user_agent";
    
    // Database attributes
    public const string DbSystem = "db.system";
    public const string DbStatement = "db.statement";
    public const string DbOperation = "db.operation.name";
    
    // Service attributes
    public const string ServiceName = "service.name";
    public const string ServiceVersion = "service.version";
    public const string ServiceNamespace = "service.namespace";
}

Identity Model Logging

JWT Token Validation Visibility

Enhanced authentication flow monitoring:

public class IdentityObservability
{
    private static readonly ActivitySource ActivitySource = 
        new("Microsoft.AspNetCore.Authentication");
    
    public async Task<AuthenticateResult> ValidateTokenAsync(string token)
    {
        using var activity = ActivitySource.StartActivity("ValidateJwtToken");
        
        activity?.SetTag("auth.token.type", "JWT");
        activity?.SetTag("auth.scheme", "Bearer");
        
        try
        {
            var result = await ValidateTokenInternalAsync(token);
            
            activity?.SetTag("auth.result", result.Succeeded ? "success" : "failure");
            activity?.SetTag("auth.principal.name", result.Principal?.Identity?.Name);
            
            if (!result.Succeeded)
            {
                activity?.SetTag("auth.failure.reason", result.Failure?.Message);
            }
            
            return result;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
}

Custom Instrumentation

Business Logic Tracing

Application-specific observability:

public class OrderService
{
    private static readonly ActivitySource ActivitySource = 
        new("ECommerce.OrderService", "1.0.0");
    
    private static readonly Counter<int> OrdersProcessed = 
        Meter.CreateCounter<int>("orders.processed");
    
    private static readonly Histogram<double> OrderProcessingTime = 
        Meter.CreateHistogram<double>("orders.processing.duration_ms");
    
    public async Task<ProcessOrderResult> ProcessOrderAsync(Order order)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        var stopwatch = Stopwatch.StartNew();
        
        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.customer_id", order.CustomerId);
        activity?.SetTag("order.total_amount", order.TotalAmount);
        
        try
        {
            // Validate inventory
            using var inventoryActivity = ActivitySource.StartActivity("ValidateInventory");
            await ValidateInventoryAsync(order.Items);
            
            // Process payment
            using var paymentActivity = ActivitySource.StartActivity("ProcessPayment");
            var paymentResult = await ProcessPaymentAsync(order);
            
            // Update inventory
            using var updateActivity = ActivitySource.StartActivity("UpdateInventory");
            await UpdateInventoryAsync(order.Items);
            
            stopwatch.Stop();
            
            OrdersProcessed.Add(1, new KeyValuePair<string, object>("status", "success"));
            OrderProcessingTime.Record(stopwatch.ElapsedMilliseconds);
            
            activity?.SetStatus(ActivityStatusCode.Ok);
            return ProcessOrderResult.Success(order.Id);
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            
            OrdersProcessed.Add(1, new KeyValuePair<string, object>("status", "error"));
            OrderProcessingTime.Record(stopwatch.ElapsedMilliseconds);
            
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.SetTag("error.type", ex.GetType().Name);
            
            throw;
        }
    }
}

.NET Aspire Observability Stack

Integrated Dashboard

Comprehensive Application View

.NET Aspire provides a unified observability dashboard that combines:

• Service topology: Visual representation of service dependencies
• Real-time metrics: Live performance indicators and health status
• Distributed traces: End-to-end request flow visualization
• Log aggregation: Centralized log viewing with correlation
• Resource monitoring: Container, database, and external service health

Dashboard Configuration

public class Program
{
    public static void Main(string[] args)
    {
        var builder = DistributedApplication.CreateBuilder(args);
        
        // Add services with automatic observability
        var apiService = builder.AddProject<Projects.ApiService>("apiservice")
                               .WithHttpEndpoint(port: 5001);
        
        var webApp = builder.AddProject<Projects.WebApp>("webapp")
                           .WithHttpEndpoint(port: 5000)
                           .WithReference(apiService);
        
        // Add databases with monitoring
        var postgres = builder.AddPostgreSQL("postgres")
                             .WithDataVolume()
                             .AddDatabase("ecommerce");
        
        var redis = builder.AddRedis("redis")
                          .WithDataVolume();
        
        // Services automatically get observability
        apiService.WithReference(postgres)
                  .WithReference(redis);
        
        builder.Build().Run();
    }
}

Service Discovery and Health Checks

Automatic Health Monitoring

Built-in health checks for all Aspire-managed services:

public class HealthCheckConfiguration
{
    public static void ConfigureHealthChecks(IServiceCollection services, 
                                           IConfiguration configuration)
    {
        services.AddHealthChecks()
                // Automatic database health checks
                .AddNpgSql(configuration.GetConnectionString("postgres"))
                .AddRedis(configuration.GetConnectionString("redis"))
                
                // HTTP endpoint health checks
                .AddUrlGroup(new Uri("https://api.example.com/health"), "external-api")
                
                // Custom business logic health checks
                .AddCheck<OrderProcessingHealthCheck>("order-processing")
                .AddCheck<PaymentServiceHealthCheck>("payment-service");
    }
}

public class OrderProcessingHealthCheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context, 
        CancellationToken cancellationToken = default)
    {
        try
        {
            // Check order processing queue
            var queueDepth = await GetOrderQueueDepthAsync();
            
            if (queueDepth > 1000)
            {
                return HealthCheckResult.Degraded(
                    $"Order queue depth is high: {queueDepth}");
            }
            
            return HealthCheckResult.Healthy($"Queue depth: {queueDepth}");
        }
        catch (Exception ex)
        {
            return HealthCheckResult.Unhealthy(
                "Order processing health check failed", ex);
        }
    }
}

Resource Monitoring

Container and Infrastructure Metrics

Automatic infrastructure monitoring for Aspire-managed resources:

public class AspireResourceMetrics
{
    // Container metrics
    public static readonly Gauge<double> ContainerCpuUsage = 
        Meter.CreateGauge<double>("container.cpu.usage_percent");
    
    public static readonly Gauge<long> ContainerMemoryUsage = 
        Meter.CreateGauge<long>("container.memory.usage_bytes");
    
    // Database connection metrics
    public static readonly Gauge<int> DatabaseConnections = 
        Meter.CreateGauge<int>("database.connections.active");
    
    // Redis cache metrics
    public static readonly Counter<int> CacheOperations = 
        Meter.CreateCounter<int>("cache.operations");
    
    public static readonly Histogram<double> CacheLatency = 
        Meter.CreateHistogram<double>("cache.operation.duration_ms");
}

Service-to-Service Communication

Automatic Trace Propagation

Seamless distributed tracing across Aspire services:

public class ApiService
{
    private readonly HttpClient httpClient;
    private readonly ILogger<ApiService> logger;
    
    public ApiService(HttpClient httpClient, ILogger<ApiService> logger)
    {
        this.httpClient = httpClient;
        this.logger = logger;
    }
    
    public async Task<ProductData> GetProductAsync(int productId)
    {
        // Trace context automatically propagated
        using var activity = ActivitySource.StartActivity("GetProduct");
        activity?.SetTag("product.id", productId);
        
        // HTTP calls automatically traced
        var response = await httpClient.GetAsync($"/products/{productId}");
        
        if (response.IsSuccessStatusCode)
        {
            var product = await response.Content.ReadFromJsonAsync<ProductData>();
            
            // Log with automatic correlation
            logger.LogInformation("Retrieved product {ProductId}: {ProductName}", 
                                productId, product.Name);
            
            return product;
        }
        
        throw new ProductNotFoundException(productId);
    }
}

Monitoring and Diagnostic Tools

Visual Studio Integration

Live Diagnostic Tools

Real-time performance monitoring during development:

public class DiagnosticToolsIntegration
{
    // CPU usage profiling
    [Conditional("DEBUG")]
    public static void StartCpuProfiling()
    {
        if (Debugger.IsAttached)
        {
            DiagnosticTools.StartCpuSampling();
        }
    }
    
    // Memory allocation tracking
    [Conditional("DEBUG")]
    public static void TrackMemoryAllocations()
    {
        if (Debugger.IsAttached)
        {
            DiagnosticTools.EnableMemoryTracking();
        }
    }
    
    // Custom event logging
    public static void LogPerformanceEvent(string eventName, 
                                         Dictionary<string, object> properties)
    {
        using var activity = DiagnosticSource.StartActivity(eventName, properties);
        
        // Event automatically appears in diagnostic tools
        DiagnosticTools.WriteEvent(eventName, properties);
    }
}

Browser DevTools Integration

Blazor WebAssembly Profiling

Native browser debugging capabilities:

// Automatic browser integration
window.blazorDiagnostics = {
    // Start performance profiling
    startProfiling: () => {
        console.profile('Blazor App Performance');
        performance.mark('profiling-start');
    },
    
    // Stop profiling and extract data
    stopProfiling: () => {
        performance.mark('profiling-end');
        performance.measure('total-execution', 'profiling-start', 'profiling-end');
        console.profileEnd('Blazor App Performance');
        
        return {
            performanceEntries: performance.getEntriesByType('measure'),
            memoryUsage: performance.memory,
            navigationTiming: performance.getEntriesByType('navigation')[0]
        };
    },
    
    // Extract memory dump
    extractMemoryDump: async () => {
        if ('memory' in performance) {
            return {
                usedJSMemory: performance.memory.usedJSMemory,
                totalJSMemory: performance.memory.totalJSMemory,
                jsMemoryLimit: performance.memory.jsMemoryLimit
            };
        }
        return null;
    }
};

Production Monitoring

Application Performance Monitoring (APM)

Integration with popular APM solutions:

public class ApmIntegration
{
    public static void ConfigureApm(IServiceCollection services, 
                                   IConfiguration configuration)
    {
        // Application Insights integration
        services.AddApplicationInsightsTelemetry(configuration);
        
        // Datadog integration
        services.AddDatadogTracing(configuration);
        
        // Custom APM provider
        services.AddOpenTelemetry()
                .WithTracing(tracing =>
                {
                    tracing.AddOtlpExporter(options =>
                    {
                        options.Endpoint = configuration["Apm:Endpoint"];
                        options.Headers = $"api-key={configuration["Apm:ApiKey"]}";
                    });
                });
    }
}

Performance Profiling and Analysis

CPU Profiling

Automatic Sampling

.NET 10 provides built-in CPU profiling capabilities:

public class CpuProfiler
{
    private static readonly ActivitySource ActivitySource = 
        new("Performance.Profiling");
    
    public static async Task<T> ProfileAsync<T>(string operationName, 
                                               Func<Task<T>> operation)
    {
        using var activity = ActivitySource.StartActivity($"Profile.{operationName}");
        var stopwatch = Stopwatch.StartNew();
        
        // Start CPU sampling
        using var cpuSampler = CpuSampler.Start();
        
        try
        {
            var result = await operation();
            
            stopwatch.Stop();
            var cpuData = cpuSampler.Stop();
            
            activity?.SetTag("duration_ms", stopwatch.ElapsedMilliseconds);
            activity?.SetTag("cpu_time_ms", cpuData.CpuTimeMilliseconds);
            activity?.SetTag("samples_collected", cpuData.SampleCount);
            
            return result;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
}

Memory Analysis

Allocation Tracking

Detailed memory allocation monitoring:

public class MemoryProfiler
{
    public static MemoryAnalysisResult AnalyzeMemoryUsage(Action operation)
    {
        // Record initial state
        var initialMemory = GC.GetTotalMemory(false);
        var initialGen0 = GC.CollectionCount(0);
        var initialGen1 = GC.CollectionCount(1);
        var initialGen2 = GC.CollectionCount(2);
        
        // Enable allocation tracking
        using var tracker = AllocationTracker.Start();
        
        // Execute operation
        operation();
        
        // Force garbage collection
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
        
        // Calculate metrics
        var finalMemory = GC.GetTotalMemory(false);
        var allocations = tracker.GetAllocations();
        
        return new MemoryAnalysisResult
        {
            TotalAllocatedBytes = allocations.TotalBytes,
            AllocationCount = allocations.Count,
            MemoryDelta = finalMemory - initialMemory,
            Gen0Collections = GC.CollectionCount(0) - initialGen0,
            Gen1Collections = GC.CollectionCount(1) - initialGen1,
            Gen2Collections = GC.CollectionCount(2) - initialGen2,
            AllocationsBy Type = allocations.GroupBy(a => a.Type)
                                           .ToDictionary(g => g.Key, g => g.Sum(a => a.Size))
        };
    }
}

Performance Benchmarking

Built-in Benchmarking Tools

Integrated performance measurement:

public class PerformanceBenchmark
{
    private static readonly Histogram<double> OperationDuration = 
        Meter.CreateHistogram<double>("benchmark.operation.duration_ms");
    
    public static async Task<BenchmarkResult> BenchmarkAsync<T>(
        string operationName,
        Func<Task<T>> operation,
        int iterations = 100)
    {
        var results = new List<double>();
        var stopwatch = new Stopwatch();
        
        // Warmup
        for (int i = 0; i < 10; i++)
        {
            await operation();
        }
        
        // Actual benchmark
        for (int i = 0; i < iterations; i++)
        {
            stopwatch.Restart();
            await operation();
            stopwatch.Stop();
            
            var duration = stopwatch.Elapsed.TotalMilliseconds;
            results.Add(duration);
            OperationDuration.Record(duration, 
                new KeyValuePair<string, object>("operation", operationName));
        }
        
        return new BenchmarkResult
        {
            OperationName = operationName,
            Iterations = iterations,
            MinDuration = results.Min(),
            MaxDuration = results.Max(),
            AverageDuration = results.Average(),
            MedianDuration = results.OrderBy(x => x).Skip(results.Count / 2).First(),
            P95Duration = results.OrderBy(x => x).Skip((int)(results.Count * 0.95)).First(),
            P99Duration = results.OrderBy(x => x).Skip((int)(results.Count * 0.99)).First()
        };
    }
}

Distributed Tracing

Cross-Service Tracing

Automatic Context Propagation

Seamless trace context flow across service boundaries:

public class DistributedTracingExample
{
    // Service A - Initiates the request
    public class OrderController : ControllerBase
    {
        private readonly IOrderService orderService;
        
        [HttpPost("/orders")]
        public async Task<IActionResult> CreateOrder([FromBody] CreateOrderRequest request)
        {
            using var activity = ActivitySource.StartActivity("CreateOrder");
            activity?.SetTag("order.customer_id", request.CustomerId);
            
            var order = await orderService.ProcessOrderAsync(request);
            return Ok(order);
        }
    }
    
    // Service B - Processes inventory
    public class InventoryService
    {
        private readonly HttpClient httpClient;
        
        public async Task<bool> CheckInventoryAsync(List<OrderItem> items)
        {
            using var activity = ActivitySource.StartActivity("CheckInventory");
            activity?.SetTag("inventory.item_count", items.Count);
            
            foreach (var item in items)
            {
                using var itemActivity = ActivitySource.StartActivity("CheckItem");
                itemActivity?.SetTag("item.sku", item.Sku);
                itemActivity?.SetTag("item.quantity", item.Quantity);
                
                // HTTP call automatically propagates trace context
                var response = await httpClient.GetAsync($"/inventory/{item.Sku}");
                var availability = await response.Content.ReadFromJsonAsync<ItemAvailability>();
                
                if (availability.Available < item.Quantity)
                {
                    itemActivity?.SetTag("inventory.sufficient", false);
                    return false;
                }
                
                itemActivity?.SetTag("inventory.sufficient", true);
            }
            
            return true;
        }
    }
}

Trace Sampling

Intelligent Sampling Strategies

Optimized trace collection for production environments:

public class TraceSamplingConfiguration
{
    public static void ConfigureSampling(TracerProviderBuilder builder)
    {
        builder.SetSampler(new CompositeSampler(
            // Always sample errors
            new ErrorSampler(),
            
            // Sample high-value operations
            new OperationBasedSampler(new Dictionary<string, double>
            {
                { "ProcessPayment", 1.0 },      // 100% sampling
                { "ProcessOrder", 0.1 },        // 10% sampling
                { "GetProduct", 0.01 }          // 1% sampling
            }),
            
            // Rate-based sampling for remaining operations
            new RateLimitingSampler(maxTracesPerSecond: 100)
        ));
    }
}

public class ErrorSampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        // Always sample traces that contain errors
        if (samplingParameters.Tags.Any(tag => 
            tag.Key == "error" || tag.Key == "exception"))
        {
            return SamplingResult.Create(SamplingDecision.RecordAndSample);
        }
        
        return SamplingResult.Create(SamplingDecision.Drop);
    }
}

Trace Analysis

Correlation and Root Cause Analysis

Advanced trace analysis capabilities:

public class TraceAnalyzer
{
    public static TraceAnalysisResult AnalyzeTrace(TraceData trace)
    {
        var spans = trace.Spans.OrderBy(s => s.StartTime).ToList();
        var rootSpan = spans.First(s => s.ParentSpanId == null);
        
        return new TraceAnalysisResult
        {
            TraceId = trace.TraceId,
            TotalDuration = rootSpan.Duration,
            SpanCount = spans.Count,
            ServiceCount = spans.Select(s => s.ServiceName).Distinct().Count(),
            ErrorCount = spans.Count(s => s.Status == SpanStatus.Error),
            
            // Critical path analysis
            CriticalPath = CalculateCriticalPath(spans),
            
            // Service dependencies
            ServiceDependencies = BuildDependencyGraph(spans),
            
            // Performance bottlenecks
            Bottlenecks = IdentifyBottlenecks(spans),
            
            // Error propagation
            ErrorPropagation = TraceErrorPropagation(spans)
        };
    }
    
    private static List<SpanSummary> CalculateCriticalPath(List<Span> spans)
    {
        // Identify the longest path through the trace
        var pathAnalysis = new Dictionary<string, TimeSpan>();
        
        foreach (var span in spans)
        {
            var pathDuration = span.Duration;
            if (span.ParentSpanId != null)
            {
                var parent = spans.First(s => s.SpanId == span.ParentSpanId);
                pathDuration += pathAnalysis.GetValueOrDefault(parent.SpanId, TimeSpan.Zero);
            }
            
            pathAnalysis[span.SpanId] = pathDuration;
        }
        
        return spans.Where(s => pathAnalysis[s.SpanId] == pathAnalysis.Values.Max())
                   .Select(s => new SpanSummary(s))
                   .ToList();
    }
}

Best Practices and Implementation Patterns

Observability-First Development

Design Principles

Building observability into the development process:

1. Instrument Early: Add observability from the beginning of development
2. Semantic Consistency: Use standardized naming conventions for metrics and traces
3. Context Propagation: Ensure trace context flows through all operations
4. Error Visibility: Make failures immediately observable and actionable
5. Performance Awareness: Monitor performance characteristics continuously

Implementation Pattern

public class ObservableService
{
    private static readonly ActivitySource ActivitySource = 
        new("MyApp.Services", "1.0.0");
    
    private static readonly Counter<int> OperationCounter = 
        Meter.CreateCounter<int>("service.operations");
    
    private static readonly Histogram<double> OperationDuration = 
        Meter.CreateHistogram<double>("service.operation.duration_ms");
    
    private readonly ILogger<ObservableService> logger;
    
    public async Task<Result<T>> ExecuteAsync<T>(string operationName, 
                                                Func<Task<T>> operation)
    {
        using var activity = ActivitySource.StartActivity(operationName);
        var stopwatch = Stopwatch.StartNew();
        
        try
        {
            logger.LogInformation("Starting operation {OperationName}", operationName);
            
            var result = await operation();
            
            stopwatch.Stop();
            OperationCounter.Add(1, new("operation", operationName), new("status", "success"));
            OperationDuration.Record(stopwatch.ElapsedMilliseconds, new("operation", operationName));
            
            activity?.SetStatus(ActivityStatusCode.Ok);
            logger.LogInformation("Completed operation {OperationName} in {Duration}ms", 
                                operationName, stopwatch.ElapsedMilliseconds);
            
            return Result<T>.Success(result);
        }
        catch (Exception ex)
        {
            stopwatch.Stop();
            OperationCounter.Add(1, new("operation", operationName), new("status", "error"));
            OperationDuration.Record(stopwatch.ElapsedMilliseconds, new("operation", operationName));
            
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.SetTag("error.type", ex.GetType().Name);
            
            logger.LogError(ex, "Operation {OperationName} failed after {Duration}ms", 
                          operationName, stopwatch.ElapsedMilliseconds);
            
            return Result<T>.Failure(ex);
        }
    }
}

Metric Design Patterns

Choosing the Right Metric Type

Guidelines for metric selection:

public class MetricDesignPatterns
{
    // Use COUNTERS for events that only increase
    public static readonly Counter<int> RequestsReceived = 
        Meter.CreateCounter<int>("http.requests.received");
    
    // Use GAUGES for values that go up and down
    public static readonly ObservableGauge<int> ActiveConnections = 
        Meter.CreateObservableGauge<int>("http.connections.active", GetActiveConnections);
    
    // Use HISTOGRAMS for distributions (latency, size, etc.)
    public static readonly Histogram<double> RequestDuration = 
        Meter.CreateHistogram<double>("http.request.duration_ms");
    
    // Use UP/DOWN COUNTERS for values that can increase or decrease
    public static readonly UpDownCounter<int> QueueDepth = 
        Meter.CreateUpDownCounter<int>("processing.queue.depth");
    
    private static int GetActiveConnections()
    {
        // Return current active connection count
        return ConnectionManager.GetActiveConnectionCount();
    }
}

Alerting and SLO/SLI Definition

Service Level Objectives

Defining measurable reliability targets:

public class ServiceLevelObjectives
{
    // SLI: Request success rate
    public static readonly Counter<int> SuccessfulRequests = 
        Meter.CreateCounter<int>("sli.requests.successful");
    
    public static readonly Counter<int> FailedRequests = 
        Meter.CreateCounter<int>("sli.requests.failed");
    
    // SLI: Request latency
    public static readonly Histogram<double> RequestLatency = 
        Meter.CreateHistogram<double>("sli.request.latency_ms");
    
    // SLO: 99.9% of requests succeed
    public static double SuccessRate => 
        SuccessfulRequests.Value / (double)(SuccessfulRequests.Value + FailedRequests.Value);
    
    // SLO: 95% of requests complete within 200ms
    public static bool LatencySloMet => 
        RequestLatency.GetPercentile(0.95) < 200;
    
    // Error budget calculation
    public static double ErrorBudget => 1.0 - SuccessRate;
    public static bool ErrorBudgetExhausted => ErrorBudget > 0.001; // 0.1% error budget
}

Real-World Scenarios

E-Commerce Application Observability

Complete Observability Implementation

End-to-end monitoring for a production e-commerce system:

public class ECommerceObservability
{
    // Business metrics
    public static readonly Counter<int> OrdersPlaced = 
        Meter.CreateCounter<int>("business.orders.placed");
    
    public static readonly Histogram<decimal> OrderValue = 
        Meter.CreateHistogram<decimal>("business.order.value");
    
    public static readonly Counter<int> PaymentAttempts = 
        Meter.CreateCounter<int>("business.payment.attempts");
    
    // Infrastructure metrics
    public static readonly Gauge<int> InventoryLevels = 
        Meter.CreateGauge<int>("inventory.levels");
    
    public static readonly Histogram<double> DatabaseQueryTime = 
        Meter.CreateHistogram<double>("database.query.duration_ms");
    
    // User experience metrics
    public static readonly Histogram<double> PageLoadTime = 
        Meter.CreateHistogram<double>("frontend.page.load_time_ms");
    
    public static readonly Counter<int> UserSessions = 
        Meter.CreateCounter<int>("user.sessions.started");
}

public class OrderProcessingObservability
{
    private static readonly ActivitySource ActivitySource = 
        new("ECommerce.OrderProcessing");
    
    public async Task<OrderResult> ProcessOrderAsync(CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.item_count", request.Items.Count);
        activity?.SetTag("order.total_value", request.TotalAmount);
        
        try
        {
            // Validate inventory with detailed tracing
            using var inventorySpan = ActivitySource.StartActivity("ValidateInventory");
            foreach (var item in request.Items)
            {
                using var itemSpan = ActivitySource.StartActivity("ValidateItem");
                itemSpan?.SetTag("item.sku", item.Sku);
                itemSpan?.SetTag("item.requested_quantity", item.Quantity);
                
                var availability = await CheckItemAvailabilityAsync(item.Sku);
                itemSpan?.SetTag("item.available_quantity", availability.Quantity);
                
                if (availability.Quantity < item.Quantity)
                {
                    throw new InsufficientInventoryException(item.Sku, item.Quantity, availability.Quantity);
                }
            }
            
            // Process payment with error handling
            using var paymentSpan = ActivitySource.StartActivity("ProcessPayment");
            paymentSpan?.SetTag("payment.amount", request.TotalAmount);
            paymentSpan?.SetTag("payment.method", request.PaymentMethod);
            
            var paymentResult = await ProcessPaymentAsync(request.Payment);
            paymentSpan?.SetTag("payment.transaction_id", paymentResult.TransactionId);
            paymentSpan?.SetTag("payment.status", paymentResult.Status);
            
            // Update inventory
            using var updateSpan = ActivitySource.StartActivity("UpdateInventory");
            await UpdateInventoryAsync(request.Items);
            
            ECommerceObservability.OrdersPlaced.Add(1, 
                new("customer_segment", GetCustomerSegment(request.CustomerId)));
            ECommerceObservability.OrderValue.Record((double)request.TotalAmount);
            
            activity?.SetStatus(ActivityStatusCode.Ok);
            return OrderResult.Success(paymentResult.TransactionId);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.SetTag("error.type", ex.GetType().Name);
            
            // Increment error metrics
            ECommerceObservability.OrdersPlaced.Add(1, 
                new("status", "failed"), 
                new("error_type", ex.GetType().Name));
            
            throw;
        }
    }
}

Microservices Communication Monitoring

Service Mesh Observability

Complete inter-service monitoring:

public class MicroservicesCommunication
{
    public class ServiceMeshMetrics
    {
        // Service-to-service communication
        public static readonly Counter<int> ServiceRequests = 
            Meter.CreateCounter<int>("service.requests");
        
        public static readonly Histogram<double> ServiceLatency = 
            Meter.CreateHistogram<double>("service.request.duration_ms");
        
        // Circuit breaker metrics
        public static readonly Gauge<int> CircuitBreakerState = 
            Meter.CreateGauge<int>("circuit_breaker.state");
        
        // Retry metrics
        public static readonly Counter<int> RetryAttempts = 
            Meter.CreateCounter<int>("service.retry.attempts");
    }
    
    public class ResilientHttpClient
    {
        private readonly HttpClient httpClient;
        private readonly CircuitBreakerService circuitBreaker;
        
        public async Task<T> CallServiceAsync<T>(string serviceName, string endpoint)
        {
            using var activity = ActivitySource.StartActivity("ServiceCall");
            activity?.SetTag("service.name", serviceName);
            activity?.SetTag("service.endpoint", endpoint);
            
            var stopwatch = Stopwatch.StartNew();
            
            try
            {
                var response = await circuitBreaker.ExecuteAsync(async () =>
                {
                    return await httpClient.GetAsync($"{serviceName}{endpoint}");
                });
                
                stopwatch.Stop();
                
                ServiceMeshMetrics.ServiceRequests.Add(1, 
                    new("source_service", "current"),
                    new("target_service", serviceName),
                    new("status", "success"));
                
                ServiceMeshMetrics.ServiceLatency.Record(stopwatch.ElapsedMilliseconds,
                    new("source_service", "current"),
                    new("target_service", serviceName));
                
                return await response.Content.ReadFromJsonAsync<T>();
            }
            catch (CircuitBreakerOpenException)
            {
                ServiceMeshMetrics.ServiceRequests.Add(1,
                    new("source_service", "current"),
                    new("target_service", serviceName),
                    new("status", "circuit_breaker_open"));
                
                throw;
            }
            catch (Exception ex)
            {
                stopwatch.Stop();
                
                ServiceMeshMetrics.ServiceRequests.Add(1,
                    new("source_service", "current"),
                    new("target_service", serviceName),
                    new("status", "error"));
                
                activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
                throw;
            }
        }
    }
}

Troubleshooting and Debugging

Common Observability Issues

Missing Traces

Debugging trace propagation problems:

public class TracePropagationDebugging
{
    public static void ValidateTraceContext(HttpContext context)
    {
        var traceParent = context.Request.Headers["traceparent"].FirstOrDefault();
        var traceState = context.Request.Headers["tracestate"].FirstOrDefault();
        
        if (string.IsNullOrEmpty(traceParent))
        {
            logger.LogWarning("No trace context found in request headers");
        }
        else
        {
            logger.LogDebug("Trace context: {TraceParent}, State: {TraceState}", 
                          traceParent, traceState);
        }
        
        // Validate current activity
        var currentActivity = Activity.Current;
        if (currentActivity == null)
        {
            logger.LogError("No current activity found - trace propagation may be broken");
        }
        else
        {
            logger.LogDebug("Current activity: {ActivityId}, Parent: {ParentId}", 
                          currentActivity.Id, currentActivity.ParentId);
        }
    }
}

Metric Collection Issues

Diagnosing missing or incorrect metrics:

public class MetricsDiagnostics
{
    public static void ValidateMetricsCollection()
    {
        var meterProvider = services.GetService<MeterProvider>();
        if (meterProvider == null)
        {
            logger.LogError("MeterProvider not registered - metrics will not be collected");
            return;
        }
        
        // Validate meter registration
        var meter = meterProvider.GetMeter("MyApp");
        if (meter == null)
        {
            logger.LogWarning("Application meter not found - verify meter name");
        }
        
        // Test metric recording
        var testCounter = meter.CreateCounter<int>("test.counter");
        testCounter.Add(1);
        
        logger.LogInformation("Metrics validation completed");
    }
}

Performance Optimization

Reducing Observability Overhead

Optimizing observability impact on application performance:

public class ObservabilityOptimization
{
    // Use sampling for high-frequency operations
    public static readonly SampledMetric<double> HighFrequencyMetric = 
        new SampledMetric<double>("high_frequency.operation", sampleRate: 0.1);
    
    // Batch metric updates
    public static readonly BatchedCounter BatchedCounter = 
        new BatchedCounter("batched.operations", flushInterval: TimeSpan.FromSeconds(5));
    
    // Conditional instrumentation
    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    public static void RecordMetricIfEnabled(string metricName, double value)
    {
        if (observabilityConfig.IsMetricEnabled(metricName))
        {
            GetMeter().CreateHistogram<double>(metricName).Record(value);
        }
    }
    
    // Async metric recording
    public static Task RecordMetricAsync(string metricName, double value)
    {
        return Task.Run(() => 
        {
            GetMeter().CreateHistogram<double>(metricName).Record(value);
        });
    }
}

This comprehensive document covers the extensive observability capabilities in .NET 10, from runtime-level enhancements to the powerful .NET Aspire observability stack. The integration of native OpenTelemetry support, enhanced metrics collection, and sophisticated diagnostic tools positions .NET 10 as a leader in application observability and monitoring.